Optimality Inequalities for Average Cost Markov Decision Processes and the Optimality of (s, S) Policies

For general state and action space Markov decision processes, we present sufficient conditions for convergence of both the optimal discounted cost value function and policies to the corresponding objects for the average costs per unit time. We extend Schäl’s [24] assumptions, guaranteeing the existence of a solution to the average cost optimality inequalities for compact action sets, to non-compact action sets. Since a stationary policy satisfying the optimality inequalities is average cost optimal, this paper provides sufficient conditions for the existence of stationary optimal policies for the average cost criterion. Inventory models are natural candidates for the application of our results. In particular, we provide straightforward proofs of the optimality of (s, S) policies for classic inventory control problems with generally distributed non-negative demand and the convergence of optimal thresholds for discounted costs to optimal thresholds for average costs per unit time as the discount factor tends to 1.